In response to this:

Summary

Backport of the noOLM / Sail Library code structure to release-4.21 with the GatewayAPIWithoutOLM feature gate disabled in all feature sets. The goal is structural alignment with 4.22 so that future bug fixes to the Sail Library path can be cleanly cherry-picked into 4.21 without massive conflicts.

The feature gate is OFF — the existing OLM path remains active. The Sail Library code is present but dormant.

Cherry-picked PRs

PR	Title	Why
#1329	OCPBUGS-70211: Fix logging for unmanaged controllers	Prerequisite — conflicts with #1354's changes to gatewayclass/controller.go
#1354	NE-2471: Replace OLM-based Istio install with Sail Library	Core change — adds istio_sail_installer.go, istio_olm.go refactor, migration.go, status.go, CRD manifests, Sail Library RBAC manifests
#1404	NE-2519: Move Sail Library to official release branch	Moves from dev Sail Library branch to official OSSM 3.3.1 release

External dependency

Repo	Branch	Why
gcs278/api@backport-GatewayAPIWithoutOLM-4.21	GatewayAPIWithoutOLM feature gate definition (disabled in all profiles)	CIO code references this gate — won't compile without it. Needs to be merged into openshift/api release-4.21 before this PR can land.

Conflicts resolved

• pkg/operator/operator.go: Added GatewayAPIWithoutOLM gate alongside existing 4.21 gates (GatewayAPI, GatewayAPIController, RouteExternalCertificate)
• pkg/operator/controller/status/controller.go: Took incoming noOLM logic (useOLM/useSailLibrary, conditional subscription listing) but wrapped in existing 4.21 GatewayAPIEnabled guard
• test/e2e/gateway_api_test.go: Kept 4.21 gatewayAPIControllerEnabled guard, added gatewayAPIWithoutOLMEnabled conditionals inside for Sail Library vs OLM test selection. Kept xcrdNames alongside new istioCRDNames
• go.mod / vendor: Added replace directives for openshift/api (fork with gate) and sail-operator (downstream fork with pkg/install)

Verification

• go build ./pkg/operator/controller/gatewayclass/... compiles
• go test ./pkg/operator/controller/gatewayclass/... passes
• Full CI (blocked on openshift/api dependency)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

In response to this:

No longer pursuing this
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

In response to this:

Some new information makes this backport attractive again.
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

In response to this:

Summary

Backport of the noOLM / Sail Library installation path (NE-2286, shipped in 4.22) to release-4.21. This resolves several fundamental OLM bugs that have no viable OLM-based workaround — most critically OCPBUGS-86778, which blocks all OSSM z-stream upgrades and prevents shipping CVE fixes.

This PR is intended to merge with the GatewayAPIWithoutOLM feature gate disabled, making it a no-op on merge. The goal is to subsequently enable the gate by default (via openshift/api) to activate the Sail Library path and resolve the OLM issues.

Cherry-picked PRs

PR	Title	Why
#1354	NE-2471: Replace OLM-based Istio install with Sail Library	Core change — adds istio_sail_installer.go, istio_olm.go refactor, migration.go, status.go, CRD manifests, Sail Library RBAC manifests
#1402	OCPBUGS-79467: Change default log level from DEBUG to INFO	Sail Library generates ~2,000 debug logs/hour; without this fix, enabling noOLM floods the logs. Only the log level change (commit 1) is cherry-picked; commit 2 references code not present on 4.21.
#1404	NE-2519: Move Sail Library to official release branch	Moves from dev Sail Library branch to official OSSM 3.3.1 release

Note: #1393 (OCPBUGS-79667: Use feature-gate annotation for Sail Library RBAC) was also a dependency but is being skipped because CVO on this release does not support the release.openshift.io/feature-gate annotation (openshift/cluster-version-operator#1273 was not backported). As a result, the Sail Library RBAC manifests use the release.openshift.io/feature-set annotation and a separate PR will be needed to remove this annotation before promoting the feature gate to GA.

Versioning

This backport does not bump the Gateway API CRDs (remain at v1.3.0) or the Istio version (remains at v1.27.3) for the noOLM code path. When the GatewayAPIWithoutOLM feature gate is enabled, the Sail Library will install Istio using the same v1.27.3 version that the OLM path currently uses. This works because the vendored Sail Library (OSSM 3.3.1) still supports Istio 1.27.3.

The GWAPI CRD bump to v1.4.1 and Istio version bump to v1.28.5 will follow separately via #1444, allowing us to validate the noOLM path independently from the version changes.

When noOLM shipped in 4.22, the OLM and noOLM versions were already aligned at 3.3.1, so version separation was not needed. On 4.21, the OLM path is on 3.2.0 — keeping both paths at the same Istio version avoids introducing conditional logic or separate deployment manifests in the backport.

Conflicts resolved

• pkg/operator/operator.go: Added GatewayAPIWithoutOLM gate alongside existing 4.21 gates (GatewayAPI, GatewayAPIController, RouteExternalCertificate)
• pkg/operator/controller/status/controller.go: Took incoming noOLM logic (useOLM/useSailLibrary, conditional subscription listing) but wrapped in existing 4.21 GatewayAPIEnabled guard
• test/e2e/gateway_api_test.go: Kept 4.21 gatewayAPIControllerEnabled guard, added gatewayAPIWithoutOLMEnabled conditionals inside for Sail Library vs OLM test selection. Kept xcrdNames alongside new istioCRDNames. Removed references to testGatewayAPIInfrastructureAnnotations, testGatewayAPIInternalLoadBalancer, and testGatewayOpenshiftConditions which were added in separate PRs not present on release-4.21.
• go.mod / vendor**: Added replace directives for openshift/api (fork with gate) and sail-operator (downstream fork with pkg/install)
• pkg/operator/controller/canary/daemonset.go (OCPBUGS-79467: Change default log level from DEBUG to INFO #1402 commit 2): Skipped — references canary cert hash variables not present on 4.21

Rollout Plan

Phase 1 — Land code (gate OFF)

• [release-4.21] OCPBUGS-85149: Backport GatewayAPIWithoutOLM feature gate as disabled api#2864 — Add FG as disabled
• [release-4.21] OCPBUGS-85149: Enable Gateway API tests on vSphere and baremetal origin#31139 — Backport Gateway API test prerequisites (baremetal/vSphere)
• [release-4.21] OCPBUGS-XXXXX: Backport noOLM Gateway API test coverage and upgrade tests origin#31232 — Backport noOLM test coverage and upgrade tests
• This PR — Sail Library code lands, gate OFF

Phase 2 — TechPreview soak

• [release-4.21] NE-2480: Promote GatewayAPIWithoutOLM feature gate to TechPreview api#2873 — FG promotion to TechPreview
• Verify CI soak in TP periodic jobs

Phase 3 — GA promotion

• [release-4.21] WIP: Remove feature-set annotations from Sail Library RBAC Manifests #1462 — Remove feature-set annotation from Sail Library RBAC
• [release-4.21] WIP: NE-2480: Promote GatewayAPIWithoutOLM feature gate to Default api#2865 — FG promotion to Default/GA, activates noOLM
• Verify CI is green

Follow-up

• OCPBUGS-84834: Bump GWAPI CRDs to v1.4.1 OSSM 3.3.1 and Istio v1.28.5 #1444 — Bump GWAPI CRDs to v1.4.1 and Istio to v1.28.5

Go Dependency Updates

Transitive dependency changes

The sail-operator (OSSM 3.3.1) brings in new transitive dependencies for Helm chart rendering (helm.sh/helm/v3), Istio utility libraries (istio.io/istio/pkg/log, pkg/ptr, pkg/slices, pkg/util/sets), and their dependency chains. These are all indirect — vendored but not imported by CIO code directly. k8s modules received a patch bump (0.34.1 → 0.34.3) from go mod tidy. Both are low risk.

controller-runtime (pinned: v0.22.5 → v0.21.0)

The sail-operator requires controller-runtime v0.22.5, but we pin back to v0.21.0 — the version CIO's own code was built and tested against on 4.21. CIO's core controller logic (client, cache, manager, controller wiring) is unchanged and continues to run against the same controller-runtime it shipped with. The sail library's install package only uses basic client.Client operations (New, Get, Create, Update) and pkg/log — all unchanged since controller-runtime v0.1. No other vendored dependency calls controller-runtime APIs.

On 4.21, this pin is not strictly required since 4.21 is already on k8s 0.34 and a patch bump poses no compatibility risk. However, on 4.20 and 4.19 the pin is essential because controller-runtime 0.22 would force a k8s minor version bump, causing incompatibilities with the frozen openshift ecosystem packages (client-go, library-go). Pinning here maintains a consistent approach across all three backport branches.

gateway-api (pinned: v1.4.1 → v1.3.0)

The sail-operator pulls in gateway-api v1.4.1, but we pin back to v1.3.0 (the original 4.21 version). The CRD manifests shipped in this release are v1.3.0, and the Go types are forward-compatible. Pinning keeps the vendored types aligned with the CRDs installed on the cluster.

Verification

• go build ./pkg/operator/controller/gatewayclass/... compiles
• go test ./pkg/operator/controller/gatewayclass/... passes
• Full CI (blocked on openshift/api dependency)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

In response to this:

Summary

Backport of the noOLM / Sail Library installation path (NE-2286, shipped in 4.22) to release-4.21. This resolves several fundamental OLM bugs that have no viable OLM-based workaround — most critically OCPBUGS-86778, which blocks all OSSM z-stream upgrades and prevents shipping CVE fixes.

This PR is intended to merge with the GatewayAPIWithoutOLM feature gate disabled, making it a no-op on merge. The goal is to subsequently enable the gate by default (via openshift/api) to activate the Sail Library path and resolve the OLM issues.

Cherry-picked PRs

PR	Title	Why
#1354	NE-2471: Replace OLM-based Istio install with Sail Library	Core change — adds istio_sail_installer.go, istio_olm.go refactor, migration.go, status.go, CRD manifests, Sail Library RBAC manifests
#1402	OCPBUGS-79467: Change default log level from DEBUG to INFO	Sail Library generates ~2,000 debug logs/hour; without this fix, enabling noOLM floods the logs. Only the log level change (commit 1) is cherry-picked; commit 2 references code not present on 4.21.
#1404	NE-2519: Move Sail Library to official release branch	Moves from dev Sail Library branch to official OSSM 3.3.1 release

Note: #1393 (OCPBUGS-79667: Use feature-gate annotation for Sail Library RBAC) was also a dependency but is being skipped because CVO on this release does not support the release.openshift.io/feature-gate annotation (openshift/cluster-version-operator#1273 was not backported). As a result, the Sail Library RBAC manifests use the release.openshift.io/feature-set annotation and a separate PR will be needed to remove this annotation before promoting the feature gate to GA.

Versioning

This backport does not bump the Gateway API CRDs (remain at v1.3.0) or the Istio version (remains at v1.27.3) for the noOLM code path. When the GatewayAPIWithoutOLM feature gate is enabled, the Sail Library will install Istio using the same v1.27.3 version that the OLM path currently uses. This works because the vendored Sail Library (OSSM 3.3.1) still supports Istio 1.27.3.

The GWAPI CRD bump to v1.4.1 and Istio version bump to v1.28.5 will follow separately via #1444, allowing us to validate the noOLM path independently from the version changes.

When noOLM shipped in 4.22, the OLM and noOLM versions were already aligned at 3.3.1, so version separation was not needed. On 4.21, the OLM path is on 3.2.0 — keeping both paths at the same Istio version avoids introducing conditional logic or separate deployment manifests in the backport.

Conflicts resolved

• pkg/operator/operator.go: Added GatewayAPIWithoutOLM gate alongside existing 4.21 gates (GatewayAPI, GatewayAPIController, RouteExternalCertificate)
• pkg/operator/controller/status/controller.go: Took incoming noOLM logic (useOLM/useSailLibrary, conditional subscription listing) but wrapped in existing 4.21 GatewayAPIEnabled guard
• test/e2e/gateway_api_test.go: Kept 4.21 gatewayAPIControllerEnabled guard, added gatewayAPIWithoutOLMEnabled conditionals inside for Sail Library vs OLM test selection. Kept xcrdNames alongside new istioCRDNames. Removed references to testGatewayAPIInfrastructureAnnotations, testGatewayAPIInternalLoadBalancer, and testGatewayOpenshiftConditions which were added in separate PRs not present on release-4.21.
• go.mod / vendor**: Added replace directives for openshift/api (fork with gate) and sail-operator (downstream fork with pkg/install)
• pkg/operator/controller/canary/daemonset.go (OCPBUGS-79467: Change default log level from DEBUG to INFO #1402 commit 2): Skipped — references canary cert hash variables not present on 4.21

Rollout Plan

Phase 1 — Land code (gate OFF)

• [release-4.21] OCPBUGS-85149: Backport GatewayAPIWithoutOLM feature gate as disabled api#2864 — Add FG as disabled
• [release-4.21] OCPBUGS-85149: Enable Gateway API tests on vSphere and baremetal origin#31139 — Backport Gateway API test prerequisites (baremetal/vSphere)
• [release-4.21] OCPBUGS-XXXXX: Backport noOLM Gateway API test coverage and upgrade tests origin#31232 — Backport noOLM test coverage and upgrade tests
• This PR — Sail Library code lands, gate OFF

Phase 2 — TechPreview soak

• [release-4.21] NE-2480: Promote GatewayAPIWithoutOLM feature gate to TechPreview api#2873 — FG promotion to TechPreview
• Verify CI soak in TP periodic jobs

Phase 3 — GA promotion

• [release-4.21] WIP: Remove feature-set annotations from Sail Library RBAC Manifests #1462 — Remove feature-set annotation from Sail Library RBAC
• [release-4.21] WIP: NE-2480: Promote GatewayAPIWithoutOLM feature gate to Default api#2865 — FG promotion to Default/GA, activates noOLM
• Verify CI is green

Follow-up

• OCPBUGS-84834: Bump GWAPI CRDs to v1.4.1 OSSM 3.3.1 and Istio v1.28.5 #1444 — Bump GWAPI CRDs to v1.4.1 and Istio to v1.28.5

Go Dependency Updates

Transitive dependency changes

The sail-operator (OSSM 3.3.1) brings in new transitive dependencies for Helm chart rendering (helm.sh/helm/v3), Istio utility libraries (istio.io/istio/pkg/log, pkg/ptr, pkg/slices, pkg/util/sets), and their dependency chains. These are all indirect — vendored but not imported by CIO code directly. k8s modules received a patch bump (0.34.1 → 0.34.3) from go mod tidy. Both are low risk.

controller-runtime (pinned: v0.22.5 → v0.21.0)

The sail-operator requires controller-runtime v0.22.5, but we pin back to v0.21.0 — the version CIO's own code was built and tested against on 4.21. CIO's core controller logic (client, cache, manager, controller wiring) is unchanged and continues to run against the same controller-runtime it shipped with. The sail library's install package only uses basic client.Client operations (New, Get, Create, Update) and pkg/log — all unchanged since controller-runtime v0.1. No other vendored dependency calls controller-runtime APIs.

On 4.21, this pin is not strictly required since 4.21 is already on k8s 0.34 and a patch bump poses no compatibility risk. However, on 4.20 and 4.19 the pin is essential because controller-runtime 0.22 would force a k8s minor version bump, causing incompatibilities with the frozen openshift ecosystem packages (client-go, library-go). Pinning here maintains a consistent approach across all three backport branches.

gateway-api (pinned: v1.4.1 → v1.3.0)

The sail-operator pulls in gateway-api v1.4.1, but we pin back to v1.3.0 (the original 4.21 version). The CRD manifests shipped in this release are v1.3.0, and the Go types are forward-compatible. Pinning keeps the vendored types aligned with the CRDs installed on the cluster.

Verification

• go build ./pkg/operator/controller/gatewayclass/... compiles
• go test ./pkg/operator/controller/gatewayclass/... passes
• Full CI (blocked on openshift/api dependency)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

In response to this:

Summary

Backport of the noOLM / Sail Library installation path (NE-2286, shipped in 4.22) to release-4.21. This resolves several fundamental OLM bugs that have no viable OLM-based workaround — most critically OCPBUGS-86778, which blocks all OSSM z-stream upgrades and prevents shipping CVE fixes.

This PR is intended to merge with the GatewayAPIWithoutOLM feature gate disabled, making it a no-op on merge. The goal is to subsequently enable the gate by default (via openshift/api) to activate the Sail Library path and resolve the OLM issues.

Cherry-picked PRs

PR	Title	Why
#1354	NE-2471: Replace OLM-based Istio install with Sail Library	Core change — adds istio_sail_installer.go, istio_olm.go refactor, migration.go, status.go, CRD manifests, Sail Library RBAC manifests
#1402	OCPBUGS-79467: Change default log level from DEBUG to INFO	Sail Library generates ~2,000 debug logs/hour; without this fix, enabling noOLM floods the logs. Only the log level change (commit 1) is cherry-picked; commit 2 references code not present on 4.21.
#1404	NE-2519: Move Sail Library to official release branch	Moves from dev Sail Library branch to official OSSM 3.3.1 release

Note: #1393 (OCPBUGS-79667: Use feature-gate annotation for Sail Library RBAC) was also a dependency but is being skipped because CVO on this release does not support the release.openshift.io/feature-gate annotation (openshift/cluster-version-operator#1273 was not backported). As a result, the Sail Library RBAC manifests use the release.openshift.io/feature-set annotation and a separate PR will be needed to remove this annotation before promoting the feature gate to GA.

Versioning

This backport does not bump the Gateway API CRDs (remain at v1.3.0) or the Istio version (remains at v1.27.3) for the noOLM code path. When the GatewayAPIWithoutOLM feature gate is enabled, the Sail Library will install Istio using the same v1.27.3 version that the OLM path currently uses. This works because the vendored Sail Library (OSSM 3.3.1) still supports Istio 1.27.3.

The GWAPI CRD bump to v1.4.1 and Istio version bump to v1.28.5 will follow separately via #1444, allowing us to validate the noOLM path independently from the version changes.

When noOLM shipped in 4.22, the OLM and noOLM versions were already aligned at 3.3.1, so version separation was not needed. On 4.21, the OLM path is on 3.2.0 — keeping both paths at the same Istio version avoids introducing conditional logic or separate deployment manifests in the backport.

Conflicts resolved

• pkg/operator/operator.go: Added GatewayAPIWithoutOLM gate alongside existing 4.21 gates (GatewayAPI, GatewayAPIController, RouteExternalCertificate)
• pkg/operator/controller/status/controller.go: Took incoming noOLM logic (useOLM/useSailLibrary, conditional subscription listing) but wrapped in existing 4.21 GatewayAPIEnabled guard
• test/e2e/gateway_api_test.go: Kept 4.21 gatewayAPIControllerEnabled guard, added gatewayAPIWithoutOLMEnabled conditionals inside for Sail Library vs OLM test selection. Kept xcrdNames alongside new istioCRDNames. Removed references to testGatewayAPIInfrastructureAnnotations, testGatewayAPIInternalLoadBalancer, and testGatewayOpenshiftConditions which were added in separate PRs not present on release-4.21.
• go.mod / vendor**: Added replace directives for openshift/api (fork with gate) and sail-operator (downstream fork with pkg/install)
• pkg/operator/controller/canary/daemonset.go (OCPBUGS-79467: Change default log level from DEBUG to INFO #1402 commit 2): Skipped — references canary cert hash variables not present on 4.21

Rollout Plan

Phase 1 — Land code (gate OFF)

• [release-4.21] OCPBUGS-85149: Backport GatewayAPIWithoutOLM feature gate as disabled api#2864 — Add FG as disabled
• [release-4.21] OCPBUGS-85149: Enable Gateway API tests on vSphere and baremetal origin#31139 — Backport Gateway API test prerequisites (baremetal/vSphere)
• [release-4.21] OCPBUGS-XXXXX: Backport noOLM Gateway API test coverage and upgrade tests origin#31232 — Backport noOLM test coverage and upgrade tests
• This PR — Sail Library code lands, gate OFF

Phase 2 — TechPreview soak

• [release-4.21] NE-2480: Promote GatewayAPIWithoutOLM feature gate to TechPreview api#2873 — FG promotion to TechPreview
• Verify CI soak in TP periodic jobs

Phase 3 — GA promotion

• [release-4.21] WIP: Remove feature-set annotations from Sail Library RBAC Manifests #1462 — Remove feature-set annotation from Sail Library RBAC
• [release-4.21] WIP: NE-2480: Promote GatewayAPIWithoutOLM feature gate to Default api#2865 — FG promotion to Default/GA, activates noOLM
• Verify CI is green

Follow-up

• OCPBUGS-84834: Bump GWAPI CRDs to v1.4.1 OSSM 3.3.1 and Istio v1.28.5 #1444 — Bump GWAPI CRDs to v1.4.1 and Istio to v1.28.5

Go Dependency Updates

Transitive dependency changes

The sail-operator (OSSM 3.3.1) brings in new transitive dependencies for Helm chart rendering (helm.sh/helm/v3), Istio utility libraries (istio.io/istio/pkg/log, pkg/ptr, pkg/slices, pkg/util/sets), and their dependency chains. These are all indirect — vendored but not imported by CIO code directly. k8s modules received a patch bump (0.34.1 → 0.34.3) from go mod tidy. Both are low risk.

controller-runtime (pinned: v0.22.5 → v0.21.0)

The sail-operator requires controller-runtime v0.22.5, but we pin back to v0.21.0 — the version CIO's own code was built and tested against on 4.21. CIO's core controller logic (client, cache, manager, controller wiring) is unchanged and continues to run against the same controller-runtime it shipped with. The sail library's install package only uses basic client.Client operations (New, Get, Create, Update) and pkg/log — all unchanged since controller-runtime v0.1. No other vendored dependency calls controller-runtime APIs.

On 4.21, this pin is not strictly required since 4.21 is already on k8s 0.34 and a patch bump poses no compatibility risk. However, on 4.20 and 4.19 the pin is essential because controller-runtime 0.22 would force a k8s minor version bump, causing incompatibilities with the frozen openshift ecosystem packages (client-go, library-go). Pinning here maintains a consistent approach across all three backport branches.

gateway-api (pinned: v1.4.1 → v1.3.0)

The sail-operator pulls in gateway-api v1.4.1, but we pin back to v1.3.0 (the original 4.21 version). The CRD manifests shipped in this release are v1.3.0, and the Go types are forward-compatible. Pinning keeps the vendored types aligned with the CRDs installed on the cluster.

Verification

• go build ./pkg/operator/controller/gatewayclass/... compiles
• go test ./pkg/operator/controller/gatewayclass/... passes
• Full CI (blocked on openshift/api dependency)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

• bug is open, matching expected state (open)
• bug target version (4.21.z) matches configured target version for branch (4.21.z)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
• release note text is set and does not match the template
• dependent bug Jira Issue OCPBUGS-86778 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
• dependent Jira Issue OCPBUGS-86778 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
• bug has dependents

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

In response to this:

/jira cherrypick OCPBUGS-79467

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

In response to this:

Summary

Backport of the noOLM / Sail Library installation path (NE-2286, shipped in 4.22) to release-4.21. This resolves several fundamental OLM bugs that have no viable OLM-based workaround — most critically OCPBUGS-86778, which blocks all OSSM z-stream upgrades and prevents shipping CVE fixes.

This PR is intended to merge with the GatewayAPIWithoutOLM feature gate disabled, making it a no-op on merge. The goal is to subsequently enable the gate by default (via openshift/api) to activate the Sail Library path and resolve the OLM issues.

Cherry-picked PRs

PR	Title	Why
#1354	NE-2471: Replace OLM-based Istio install with Sail Library	Core change — adds istio_sail_installer.go, istio_olm.go refactor, migration.go, status.go, CRD manifests, Sail Library RBAC manifests
#1402	OCPBUGS-79467: Change default log level from DEBUG to INFO	Sail Library generates ~2,000 debug logs/hour; without this fix, enabling noOLM floods the logs. Only the log level change (commit 1) is cherry-picked; commit 2 references code not present on 4.21.
#1404	NE-2519: Move Sail Library to official release branch	Moves from dev Sail Library branch to official OSSM 3.3.1 release

Note: #1393 (OCPBUGS-79667: Use feature-gate annotation for Sail Library RBAC) was also a dependency but is being skipped because CVO on this release does not support the release.openshift.io/feature-gate annotation (openshift/cluster-version-operator#1273 was not backported). As a result, the Sail Library RBAC manifests use the release.openshift.io/feature-set annotation and a separate PR will be needed to remove this annotation before promoting the feature gate to GA.

Versioning

This backport does not bump the Gateway API CRDs (remain at v1.3.0) or the Istio version (remains at v1.27.3) for the noOLM code path. When the GatewayAPIWithoutOLM feature gate is enabled, the Sail Library will install Istio using the same v1.27.3 version that the OLM path currently uses. This works because the vendored Sail Library (OSSM 3.3.1) still supports Istio 1.27.3.

The GWAPI CRD bump to v1.4.1 and Istio version bump to v1.28.5 will follow separately via #1444, allowing us to validate the noOLM path independently from the version changes.

When noOLM shipped in 4.22, the OLM and noOLM versions were already aligned at 3.3.1, so version separation was not needed. On 4.21, the OLM path is on 3.2.0 — keeping both paths at the same Istio version avoids introducing conditional logic or separate deployment manifests in the backport.

Conflicts resolved

• pkg/operator/operator.go: Added GatewayAPIWithoutOLM gate alongside existing 4.21 gates (GatewayAPI, GatewayAPIController, RouteExternalCertificate)
• pkg/operator/controller/status/controller.go: Took incoming noOLM logic (useOLM/useSailLibrary, conditional subscription listing) but wrapped in existing 4.21 GatewayAPIEnabled guard
• test/e2e/gateway_api_test.go: Kept 4.21 gatewayAPIControllerEnabled guard, added gatewayAPIWithoutOLMEnabled conditionals inside for Sail Library vs OLM test selection. Kept xcrdNames alongside new istioCRDNames. Removed references to testGatewayAPIInfrastructureAnnotations, testGatewayAPIInternalLoadBalancer, and testGatewayOpenshiftConditions which were added in separate PRs not present on release-4.21.
• go.mod / vendor**: Added replace directives for openshift/api (fork with gate) and sail-operator (downstream fork with pkg/install)
• pkg/operator/controller/canary/daemonset.go (OCPBUGS-79467: Change default log level from DEBUG to INFO #1402 commit 2): Skipped — references canary cert hash variables not present on 4.21

Rollout Plan

Phase 1 — Land code (gate OFF)

• [release-4.21] OCPBUGS-85149: Backport GatewayAPIWithoutOLM feature gate as disabled api#2864 — Add FG as disabled
• [release-4.21] OCPBUGS-85149: Enable Gateway API tests on vSphere and baremetal origin#31139 — Backport Gateway API test prerequisites (baremetal/vSphere)
• [release-4.21] OCPBUGS-XXXXX: Backport noOLM Gateway API test coverage and upgrade tests origin#31232 — Backport noOLM test coverage and upgrade tests
• This PR — Sail Library code lands, gate OFF

Phase 2 — TechPreview soak

• [release-4.21] NE-2480: Promote GatewayAPIWithoutOLM feature gate to TechPreview api#2873 — FG promotion to TechPreview
• Verify CI soak in TP periodic jobs

Phase 3 — GA promotion

• [release-4.21] WIP: Remove feature-set annotations from Sail Library RBAC Manifests #1462 — Remove feature-set annotation from Sail Library RBAC
• [release-4.21] WIP: NE-2480: Promote GatewayAPIWithoutOLM feature gate to Default api#2865 — FG promotion to Default/GA, activates noOLM
• Verify CI is green

Follow-up

• OCPBUGS-84834: Bump GWAPI CRDs to v1.4.1 OSSM 3.3.1 and Istio v1.28.5 #1444 — Bump GWAPI CRDs to v1.4.1 and Istio to v1.28.5

Go Dependency Updates

Transitive dependency changes

The sail-operator (OSSM 3.3.1) brings in new transitive dependencies for Helm chart rendering (helm.sh/helm/v3), Istio utility libraries (istio.io/istio/pkg/log, pkg/ptr, pkg/slices, pkg/util/sets), and their dependency chains. These are all indirect — vendored but not imported by CIO code directly. k8s modules received a patch bump (0.34.1 → 0.34.3) from go mod tidy. Both are low risk.

controller-runtime (pinned: v0.22.5 → v0.21.0)

The sail-operator requires controller-runtime v0.22.5, but we pin back to v0.21.0 — the version CIO's own code was built and tested against on 4.21. CIO's core controller logic (client, cache, manager, controller wiring) is unchanged and continues to run against the same controller-runtime it shipped with. The sail library's install package only uses basic client.Client operations (New, Get, Create, Update) and pkg/log — all unchanged since controller-runtime v0.1. No other vendored dependency calls controller-runtime APIs.

On 4.21, this pin is not strictly required since 4.21 is already on k8s 0.34 and a patch bump poses no compatibility risk. However, on 4.20 and 4.19 the pin is essential because controller-runtime 0.22 would force a k8s minor version bump, causing incompatibilities with the frozen openshift ecosystem packages (client-go, library-go). Pinning here maintains a consistent approach across all three backport branches.

gateway-api (pinned: v1.4.1 → v1.3.0)

The sail-operator pulls in gateway-api v1.4.1, but we pin back to v1.3.0 (the original 4.21 version). The CRD manifests shipped in this release are v1.3.0, and the Go types are forward-compatible. Pinning keeps the vendored types aligned with the CRDs installed on the cluster.

Verification

• go build ./pkg/operator/controller/gatewayclass/... compiles
• go test ./pkg/operator/controller/gatewayclass/... passes
• Full CI (blocked on openshift/api dependency)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

• bug is open, matching expected state (open)
• bug target version (4.21.z) matches configured target version for branch (4.21.z)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
• release note text is set and does not match the template
• dependent bug Jira Issue OCPBUGS-86778 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
• dependent Jira Issue OCPBUGS-86778 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
• bug has dependents

In response to this:

Summary

Backport of the noOLM / Sail Library installation path (NE-2286, shipped in 4.22) to release-4.21. This resolves several fundamental OLM bugs that have no viable OLM-based workaround — most critically OCPBUGS-86778, which blocks all OSSM z-stream upgrades and prevents shipping CVE fixes.

This PR is intended to merge with the GatewayAPIWithoutOLM feature gate disabled, making it a no-op on merge. The goal is to subsequently enable the gate by default (via openshift/api) to activate the Sail Library path and resolve the OLM issues.

Cherry-picked PRs

PR	Title	Why
#1354	NE-2471: Replace OLM-based Istio install with Sail Library	Core change — adds istio_sail_installer.go, istio_olm.go refactor, migration.go, status.go, CRD manifests, Sail Library RBAC manifests
#1402	OCPBUGS-79467: Change default log level from DEBUG to INFO	Sail Library generates ~2,000 debug logs/hour; without this fix, enabling noOLM floods the logs. Only the log level change (commit 1) is cherry-picked; commit 2 references code not present on 4.21.
#1404	NE-2519: Move Sail Library to official release branch	Moves from dev Sail Library branch to official OSSM 3.3.1 release

Note: #1393 (OCPBUGS-79667: Use feature-gate annotation for Sail Library RBAC) was also a dependency but is being skipped because CVO on this release does not support the release.openshift.io/feature-gate annotation (openshift/cluster-version-operator#1273 was not backported). As a result, the Sail Library RBAC manifests use the release.openshift.io/feature-set annotation and a separate PR will be needed to remove this annotation before promoting the feature gate to GA.

Versioning

This backport does not bump the Gateway API CRDs (remain at v1.3.0) or the Istio version (remains at v1.27.3) for the noOLM code path. When the GatewayAPIWithoutOLM feature gate is enabled, the Sail Library will install Istio using the same v1.27.3 version that the OLM path currently uses. This works because the vendored Sail Library (OSSM 3.3.1) still supports Istio 1.27.3.

The GWAPI CRD bump to v1.4.1 and Istio version bump to v1.28.5 will follow separately via #1444, allowing us to validate the noOLM path independently from the version changes.

When noOLM shipped in 4.22, the OLM and noOLM versions were already aligned at 3.3.1, so version separation was not needed. On 4.21, the OLM path is on 3.2.0 — keeping both paths at the same Istio version avoids introducing conditional logic or separate deployment manifests in the backport.

Conflicts resolved

• pkg/operator/operator.go: Added GatewayAPIWithoutOLM gate alongside existing 4.21 gates (GatewayAPI, GatewayAPIController, RouteExternalCertificate)
• pkg/operator/controller/status/controller.go: Took incoming noOLM logic (useOLM/useSailLibrary, conditional subscription listing) but wrapped in existing 4.21 GatewayAPIEnabled guard
• test/e2e/gateway_api_test.go: Kept 4.21 gatewayAPIControllerEnabled guard, added gatewayAPIWithoutOLMEnabled conditionals inside for Sail Library vs OLM test selection. Kept xcrdNames alongside new istioCRDNames. Removed references to testGatewayAPIInfrastructureAnnotations, testGatewayAPIInternalLoadBalancer, and testGatewayOpenshiftConditions which were added in separate PRs not present on release-4.21.
• go.mod / vendor**: Added replace directives for openshift/api (fork with gate) and sail-operator (downstream fork with pkg/install)
• pkg/operator/controller/canary/daemonset.go (OCPBUGS-79467: Change default log level from DEBUG to INFO #1402 commit 2): Skipped — references canary cert hash variables not present on 4.21

Rollout Plan

Phase 1 — Land code (gate OFF)

• [release-4.21] OCPBUGS-85149: Backport GatewayAPIWithoutOLM feature gate as disabled api#2864 — Add FG as disabled
• [release-4.21] OCPBUGS-85149: Enable Gateway API tests on vSphere and baremetal origin#31139 — Backport Gateway API test prerequisites (baremetal/vSphere)
• [release-4.21] OCPBUGS-XXXXX: Backport noOLM Gateway API test coverage and upgrade tests origin#31232 — Backport noOLM test coverage and upgrade tests
• This PR — Sail Library code lands, gate OFF

Phase 2 — TechPreview soak

• [release-4.21] NE-2480: Promote GatewayAPIWithoutOLM feature gate to TechPreview api#2873 — FG promotion to TechPreview
• Verify CI soak in TP periodic jobs

Phase 3 — GA promotion

• [release-4.21] WIP: Remove feature-set annotations from Sail Library RBAC Manifests #1462 — Remove feature-set annotation from Sail Library RBAC
• [release-4.21] WIP: NE-2480: Promote GatewayAPIWithoutOLM feature gate to Default api#2865 — FG promotion to Default/GA, activates noOLM
• Verify CI is green

Follow-up

• OCPBUGS-84834: Bump GWAPI CRDs to v1.4.1 OSSM 3.3.1 and Istio v1.28.5 #1444 — Bump GWAPI CRDs to v1.4.1 and Istio to v1.28.5

Go Dependency Updates

Transitive dependency changes

The sail-operator (OSSM 3.3.1) brings in new transitive dependencies for Helm chart rendering (helm.sh/helm/v3), Istio utility libraries (istio.io/istio/pkg/log, pkg/ptr, pkg/slices, pkg/util/sets), and their dependency chains. These are all indirect — vendored but not imported by CIO code directly. k8s modules received a patch bump (0.34.1 → 0.34.3) from go mod tidy. Both are low risk.

controller-runtime (pinned: v0.22.5 → v0.21.0)

The sail-operator requires controller-runtime v0.22.5, but we pin back to v0.21.0 — the version CIO's own code was built and tested against on 4.21. CIO's core controller logic (client, cache, manager, controller wiring) is unchanged and continues to run against the same controller-runtime it shipped with. The sail library's install package only uses basic client.Client operations (New, Get, Create, Update) and pkg/log — all unchanged since controller-runtime v0.1. No other vendored dependency calls controller-runtime APIs.

On 4.21, this pin is not strictly required since 4.21 is already on k8s 0.34 and a patch bump poses no compatibility risk. However, on 4.20 and 4.19 the pin is essential because controller-runtime 0.22 would force a k8s minor version bump, causing incompatibilities with the frozen openshift ecosystem packages (client-go, library-go). Pinning here maintains a consistent approach across all three backport branches.

gateway-api (pinned: v1.4.1 → v1.3.0)

The sail-operator pulls in gateway-api v1.4.1, but we pin back to v1.3.0 (the original 4.21 version). The CRD manifests shipped in this release are v1.3.0, and the Go types are forward-compatible. Pinning keeps the vendored types aligned with the CRDs installed on the cluster.

Verification

• go build ./pkg/operator/controller/gatewayclass/... compiles
• go test ./pkg/operator/controller/gatewayclass/... passes
• Full CI (blocked on openshift/api dependency)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

• bug is open, matching expected state (open)
• bug target version (4.21.z) matches configured target version for branch (4.21.z)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
• release note text is set and does not match the template
• dependent bug Jira Issue OCPBUGS-86778 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
• dependent Jira Issue OCPBUGS-86778 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
• bug has dependents

• bug is open, matching expected state (open)
• bug target version (4.21.z) matches configured target version for branch (4.21.z)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
• release note text is set and does not match the template
• dependent bug Jira Issue OCPBUGS-79467 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
• dependent Jira Issue OCPBUGS-79467 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
• bug has dependents

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

• bug is open, matching expected state (open)
• bug target version (4.21.z) matches configured target version for branch (4.21.z)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
• release note text is set and does not match the template
• dependent bug Jira Issue OCPBUGS-86778 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
• dependent Jira Issue OCPBUGS-86778 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
• bug has dependents

• bug is open, matching expected state (open)
• bug target version (4.21.z) matches configured target version for branch (4.21.z)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
• release note text is set and does not match the template
• dependent bug Jira Issue OCPBUGS-79467 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
• dependent Jira Issue OCPBUGS-79467 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
• bug has dependents

• bug is open, matching expected state (open)
• bug target version (4.21.z) matches configured target version for branch (4.21.z)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
• release note text is set and does not match the template
• dependent bug Jira Issue OCPBUGS-76609 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
• dependent Jira Issue OCPBUGS-76609 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
• bug has dependents

In response to this:

Summary

Backport of the noOLM / Sail Library installation path (NE-2286, shipped in 4.22) to release-4.21. This resolves several fundamental OLM bugs that have no viable OLM-based workaround — most critically OCPBUGS-86778, which blocks all OSSM z-stream upgrades and prevents shipping CVE fixes.

This PR is intended to merge with the GatewayAPIWithoutOLM feature gate disabled, making it a no-op on merge. The goal is to subsequently enable the gate by default (via openshift/api) to activate the Sail Library path and resolve the OLM issues.

Cherry-picked PRs

PR	Title	Why
#1354	NE-2471: Replace OLM-based Istio install with Sail Library	Core change — adds istio_sail_installer.go, istio_olm.go refactor, migration.go, status.go, CRD manifests, Sail Library RBAC manifests
#1402	OCPBUGS-79467: Change default log level from DEBUG to INFO	Sail Library generates ~2,000 debug logs/hour; without this fix, enabling noOLM floods the logs. Only the log level change (commit 1) is cherry-picked; commit 2 references code not present on 4.21.
#1404	NE-2519: Move Sail Library to official release branch	Moves from dev Sail Library branch to official OSSM 3.3.1 release

Note: #1393 (OCPBUGS-79667: Use feature-gate annotation for Sail Library RBAC) was also a dependency but is being skipped because CVO on this release does not support the release.openshift.io/feature-gate annotation (openshift/cluster-version-operator#1273 was not backported). As a result, the Sail Library RBAC manifests use the release.openshift.io/feature-set annotation and a separate PR will be needed to remove this annotation before promoting the feature gate to GA.

Versioning

This backport does not bump the Gateway API CRDs (remain at v1.3.0) or the Istio version (remains at v1.27.3) for the noOLM code path. When the GatewayAPIWithoutOLM feature gate is enabled, the Sail Library will install Istio using the same v1.27.3 version that the OLM path currently uses. This works because the vendored Sail Library (OSSM 3.3.1) still supports Istio 1.27.3.

The GWAPI CRD bump to v1.4.1 and Istio version bump to v1.28.5 will follow separately via #1444, allowing us to validate the noOLM path independently from the version changes.

When noOLM shipped in 4.22, the OLM and noOLM versions were already aligned at 3.3.1, so version separation was not needed. On 4.21, the OLM path is on 3.2.0 — keeping both paths at the same Istio version avoids introducing conditional logic or separate deployment manifests in the backport.

Conflicts resolved

• pkg/operator/operator.go: Added GatewayAPIWithoutOLM gate alongside existing 4.21 gates (GatewayAPI, GatewayAPIController, RouteExternalCertificate)
• pkg/operator/controller/status/controller.go: Took incoming noOLM logic (useOLM/useSailLibrary, conditional subscription listing) but wrapped in existing 4.21 GatewayAPIEnabled guard
• test/e2e/gateway_api_test.go: Kept 4.21 gatewayAPIControllerEnabled guard, added gatewayAPIWithoutOLMEnabled conditionals inside for Sail Library vs OLM test selection. Kept xcrdNames alongside new istioCRDNames. Removed references to testGatewayAPIInfrastructureAnnotations, testGatewayAPIInternalLoadBalancer, and testGatewayOpenshiftConditions which were added in separate PRs not present on release-4.21.
• go.mod / vendor**: Added replace directives for openshift/api (fork with gate) and sail-operator (downstream fork with pkg/install)
• pkg/operator/controller/canary/daemonset.go (OCPBUGS-79467: Change default log level from DEBUG to INFO #1402 commit 2): Skipped — references canary cert hash variables not present on 4.21

Rollout Plan

Phase 1 — Land code (gate OFF)

• [release-4.21] OCPBUGS-85149: Backport GatewayAPIWithoutOLM feature gate as disabled api#2864 — Add FG as disabled
• [release-4.21] OCPBUGS-85149: Enable Gateway API tests on vSphere and baremetal origin#31139 — Backport Gateway API test prerequisites (baremetal/vSphere)
• [release-4.21] OCPBUGS-XXXXX: Backport noOLM Gateway API test coverage and upgrade tests origin#31232 — Backport noOLM test coverage and upgrade tests
• This PR — Sail Library code lands, gate OFF

Phase 2 — TechPreview soak

• [release-4.21] NE-2480: Promote GatewayAPIWithoutOLM feature gate to TechPreview api#2873 — FG promotion to TechPreview
• Verify CI soak in TP periodic jobs

Phase 3 — GA promotion

• [release-4.21] WIP: Remove feature-set annotations from Sail Library RBAC Manifests #1462 — Remove feature-set annotation from Sail Library RBAC
• [release-4.21] WIP: NE-2480: Promote GatewayAPIWithoutOLM feature gate to Default api#2865 — FG promotion to Default/GA, activates noOLM
• Verify CI is green

Follow-up

• OCPBUGS-84834: Bump GWAPI CRDs to v1.4.1 OSSM 3.3.1 and Istio v1.28.5 #1444 — Bump GWAPI CRDs to v1.4.1 and Istio to v1.28.5

Go Dependency Updates

Transitive dependency changes

The sail-operator (OSSM 3.3.1) brings in new transitive dependencies for Helm chart rendering (helm.sh/helm/v3), Istio utility libraries (istio.io/istio/pkg/log, pkg/ptr, pkg/slices, pkg/util/sets), and their dependency chains. These are all indirect — vendored but not imported by CIO code directly. k8s modules received a patch bump (0.34.1 → 0.34.3) from go mod tidy. Both are low risk.

controller-runtime (pinned: v0.22.5 → v0.21.0)

The sail-operator requires controller-runtime v0.22.5, but we pin back to v0.21.0 — the version CIO's own code was built and tested against on 4.21. CIO's core controller logic (client, cache, manager, controller wiring) is unchanged and continues to run against the same controller-runtime it shipped with. The sail library's install package only uses basic client.Client operations (New, Get, Create, Update) and pkg/log — all unchanged since controller-runtime v0.1. No other vendored dependency calls controller-runtime APIs.

On 4.21, this pin is not strictly required since 4.21 is already on k8s 0.34 and a patch bump poses no compatibility risk. However, on 4.20 and 4.19 the pin is essential because controller-runtime 0.22 would force a k8s minor version bump, causing incompatibilities with the frozen openshift ecosystem packages (client-go, library-go). Pinning here maintains a consistent approach across all three backport branches.

gateway-api (pinned: v1.4.1 → v1.3.0)

The sail-operator pulls in gateway-api v1.4.1, but we pin back to v1.3.0 (the original 4.21 version). The CRD manifests shipped in this release are v1.3.0, and the Go types are forward-compatible. Pinning keeps the vendored types aligned with the CRDs installed on the cluster.

Verification

• go build ./pkg/operator/controller/gatewayclass/... compiles
• go test ./pkg/operator/controller/gatewayclass/... passes
• Full CI (blocked on openshift/api dependency)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.